Optimal Learning via the Fourier Transform for Sums of Independent Integer Random Variables

We study the structure and learnability of sums of independent integer random variables (SIIRVs). For k ∈ Z+, a k-SIIRV of order n ∈ Z+ is the probability distribution of the sum of n mutually independent random variables each supported on {0, 1, . . . , k − 1}. We denote by Sn,k the set of all k-SIIRVs of order n. How many samples are required to learn an arbitrary distribution in Sn,k? In this paper, we tightly characterize the sample and computational complexity of this problem. More precisely, we design a computationally efficient algorithm that uses Õ(k/ǫ) samples, and learns an arbitrary k-SIIRV within error ǫ, in total variation distance. Moreover, we show that the optimal sample complexity of this learning problem is Θ((k/ǫ) √ log(1/ǫ)), i.e., we prove an upper bound and a matching information-theoretic lower bound. Our algorithm proceeds by learning the Fourier transform of the target k-SIIRV in its effective support. Its correctness relies on the approximate sparsity of the Fourier transform of k-SIIRVs – a structural property that we establish, roughly stating that the Fourier transform of k-SIIRVs has small magnitude outside a small set. Along the way we prove several new structural results about k-SIIRVs. As one of our main structural contributions, we give an efficient algorithm to construct a sparse proper ǫ-cover for Sn,k, in total variation distance. We also obtain a novel geometric characterization of the space of k-SIIRVs. Our characterization allows us to prove a tight lower bound on the size of ǫ-covers for Sn,k – establishing that our cover upper bound is optimal – and is the key ingredient in our tight sample complexity lower bound. Our approach of exploiting the sparsity of the Fourier transform in distribution learning is general, and has recently found additional applications. In a subsequent work [DKS15a], we use a generalization of this idea (in higher dimensions) to obtain the first efficient learning algorithm for Poisson multinomial distributions. In [DKS15b], we build on this approach to obtain the fastest known proper learning algorithm for Poisson binomial distributions (2-SIIRVs). Supported by EPSRC grant EP/L021749/1 and a Marie Curie Career Integration grant. Some of this work was performed while visiting the University of Edinburgh. Supported by EPSRC grant EP/L021749/1.